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Abstract.  The  paper  describes  a  multimodal  guidance  system  whose 
aim  is  to  improve  manual  skills  of  people  with  specific  disorders,  such 
as  Down  syndrome,  mental  retardation,  blind,  autistic,  etc.  The  multi¬ 
modal  guidance  system  provides  assistance  in  the  execution  of  2D  tasks 
as  for  example:  sketching,  hatching  and  cutting  operations  through  hap¬ 
tic  and  sound  interactions.  The  haptic  technology  provides  the  virtual 
path  of  2D  shapes  through  the  point-based  approach,  while  sound  tech¬ 
nology  provides  some  audio  feedback  inputs  about  his  or  her  actions 
while  performing  a  manual  task  as  for  example:  start  and/or  finish  an 
sketch;  some  alarms  related  to  the  hand’s  velocity  while  sketching  and 
filling  or  cutting  operations.  Unskilled  people  use  these  interfaces  in  their 
educational  environment. 
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1  Introduction 

The  multimodal  system  presented  in  this  paper,  consisting  in  a  combination  of 
haptic  and  sound  technologies,  aims  to  be  a  step  forward  in  the  field  of  mul¬ 
timodal  devices  for  supporting  unskilled  people  to  improve  their  skills  and  in 
the  assessment  of  manual  activities.  Sketching,  hatching  and  cutting  tasks  are 
assisted  through  the  haptic  guidance  device.  A  Magnetic  Geometry  Effect  assist 
the  users  hand  movement,  which  is  a  sort  of  magnet  or  spring  effect  attracting 
the  hand  towards  the  ideal  path.  The  drawn  shape  can  also  be  physically  pro¬ 
duced  as  a  piece  of  polystyrene  foam.  The  cutting  operation  is  performed  by 
using  a  hot  wire  tool,  which  is  linked  to  the  haptic  device.  In  addition,  several 
sound  metaphors  are  explored.  These  sounds  are  used  to  give  information  related 
to  the  starting  and/or  finishing  of  an  activity  (e.g.  Sound  A  means,  starting  to 
cut  while  sound  B  means  Stop  the  cutting  activity  and  Sound  C,  can  be  used  to 
indicate  that  the  velocity  in  the  cutting  task  is  the  most  performing,  etc.) 

The  objective  of  the  multimodal  system  is  to  experiment  a  new  tool  that  al¬ 
lows  unskilled  people  to  perform  the  assessment  of  manual  skills  in  an  intuitive, 
natural  and  easy  manner.  This  group  of  people  have  demonstrated  that  with 
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early  intervention  programs,  their  possibilities  of  having  a  better  life  are  grow¬ 
ing  as  well.  Unfortunately  today  practice  to  control  the  motion  control  and  skill 
improvements  need  to  be  done  with  continuous  assistance  given  by  care  assis¬ 
tants,  so  greatly  limiting  the  possibility  to  exploit  the  great  potential  that  those 
people  have.  However,  there  is  still  much  more  opportunity  for  developing  tools 
to  help  and  support  them  for  improving  both,  their  productivity  and  indepen¬ 
dence  in  the  workplace.  This  can  be  done,  by  providing  tools  to  employers  and 
employees  to  assist  them  to  maximize  the  operational  capacity.  The  multimodal 
system  is  designed  as  a  tool  that  supports  manipulation  and  actions  needing  a 
very  limited  support  provided  by  care  assistants,  for  ensuring  the  integration 
and  independence  in  the  workforce. 


2  Related  Work 

The  research  concerning  haptic  technology  has  increased  rapidly  in  the  last  few 
years,  and  results  have  shown  the  significant  role  that  haptic  feedback  plays  in 
several  fields,  including  rehabilitation.  The  interaction  is  enriched  by  the  use 
of  the  sense  of  touch,  so  that  also  visually  impaired  users  can  identify  virtual 
objects  and  perceive  their  shape  and  texture.  Within  the  field  of  virtual  reality 
environments  and  simulation  tools,  the  sense  of  touch  is  provided  by  haptic  in¬ 
terfaces  pQ .  Haptic  interfaces  are  based  on  devices  that  present  tactile  and  force 
feedback  to  a  human  user  who  is  interacting  with  a  simulated  object  via  a  com¬ 
puter  [2]  in  order  to  feel  the  virtual  object  properties  (i.e.,  texture,  compliance  or 
shape).  Examples  include  devices  that  provide  robotic-assisted  repetitive  motion 
|3].  There  are,  however,  very  few  assisted  applications  for  unskilled  people  which 
support  them  in  a  specific  employment  role. 

Haptic  interface  technology  can  enable  individuals  who  are  blind  to  expand 
their  knowledge  by  using  an  artificially  made  reality  built  on  haptic  and  audio 
feedback.  Research  on  the  implementation  of  haptic  technologies  within  Virtual 
Environments  has  reported  the  potential  for  supporting  the  development  of  cog¬ 
nitive  models  of  navigation  and  spatial  knowledge  with  sighted  people  a,  a, 
0,  m  and  with  people  who  are  blind  as  well  [8].  Audio  feedbacks  have  been 
used  coupled  to  haptic  feedback  in  several  fields.  In  [T  for  example  have  been 
used  to  train  ophthalmic  surgeon  on  complex  optical  operation. 

We  have  used  the  Phantom  Desktop  device  that  is  available  in  our  laboratory 
in  order  to  show  and  prove  the  concept  of  the  multimodal  guidance  system.  In 
fact,  at  the  current  development  stage,  the  system  is  a  prototype  that  requires 
to  be  engineered  in  order  to  built  up  a  very  low  cost  system;  a  goal  that  appear 
very  realistic.  However,  we  have  planned  in  the  engineering  process  to  integrate 
the  force  feedback  directly  in  our  guidance  system  instead  of  using  a  commercial 
phantom  device,  taking  into  account  several  previous  work  uni,  m  in  which  the 
force  feedback  has  been  integrated  directly  into  the  pantograph  mechanism.  The 
guidance  haptic  device  concept  has  been  described  in  [12] .  This  paper  focuses 
on  the  multimodal  approach  through  the  sound  metaphors  and  the  results  of 
testing  the  first  prototype. 
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3  System  Description 

The  multimodal  guidance  system  allows  the  initial  definition  of  a  set  of  geometric 
shapes  that  the  users  will  draw,  and  physically  produce  thanks  to  the  cutting 
system  in  an  assisted  way.  Figure  [T]  provides  a  schematic  view  of  the  system’s 
architecture.  The  shapes  are  initially  generated  through  the  use  of  a  generic  CAD 
tool.  The  shapes  are  saved  in  the  VRML  format,  which  is  a  standard  file  format 
for  representing  3-dimensional  interactive  vector  graphics.  This  file  includes  the 
IndexedFace  set  list,  which  represents  the  3D  shape  formed  by  constructing  faces 
(polygons),  and  the  Coordinate  point  list,  which  contains  the  coordinate  of  each 
single  node  that  defines  the  3D  vertices  of  the  shape.  Finally,  these  data  are 
imported  in  the  H3DAPI  software  that  is  used  for  rendering  the  haptic  guiding 
path,  on  the  basis  of  the  geometry  of  the  shape. 


PHYSICAL  SHAPE 


Unskilled  User 


Fig.  1.  Multimodal  guidance  system  architecture 

This  software  was  chosen  because  is  an  open  source  platform  that  allows  to 
handle  both  graphics  and  haptic  data.  The  software  also  allows  to  easily  manage 
the  Magnetic  Surface  constraint,  which  provides  a  force  on  the  haptic  device 
based  on  a  given  distance  from  a  virtual  surface.  In  this  way,  a  snap  constraint 
is  applied  allowing  to  control  and  vary  as  needed  the  stiffness  and  damping 
constraints.  The  snap  distance  is  a  parameter  that  defines  the  outward  distance 
for  the  application  of  the  attracting  effect. 

4  The  Concept  from  the  Disabled  People  Side 

Figure  |2]  provides  the  isometric  view  of  both,  the  CAD  concept  and  the  real 
prototype  of  the  multimodal  system  seen  from  the  users  side.  The  user  is  sitting 
in  front  of  the  multimodal  system  as  can  be  seen  from  Figure  |2]-a,  and  then 
by  handling  the  stylus  (3)  tries  to  follow  the  physical  template  (1)  in  order  to 
perform  the  2D  drafting  tasks.  This  task  is  driven  under  the  operators  movement 
and  assisted  by  the  Magnetic  Geometry  Effect  (MGE).  When  this  option  is 
enabled,  a  spring  force  tries  to  pull  the  tip  of  the  stylus  (3)  of  the  haptic  device 
towards  the  virtual  path.  In  fact,  this  effect  is  used  in  order  to  assist  the  users 
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hand  (Figure|2]-b,  Figure|2]-c  and  Figure|2]-d).  While  user  follows  the  2D  template 
(1)  by  using  the  haptic  guidance,  the  wire  tool  (6)  when  activated  will  cut  the 
polystyrene  foam  (2).  Figure  |2]-e  shows  the  physical  prototype  and  the  users 
hand  holding  the  tool. 


Fig.  2.  The  concept  from  the  disable  people  side 


The  lengths  of  the  links  in  the  Phantom  device  (1)  determine  the  mechanisms 
kinematic  properties,  such  as  workspace  and  manipulability. 

5  The  Concept  from  the  Care  Assistants  Side 

We  have  designed  a  Graphic  User  Interface  in  order  to  involve  teachers  and  care 
assistants  during  the  test  phase.  In  this  way,  we  are  also  providing  some  training 
for  the  use  of  the  system.  Figure  |3]-a  shows  the  teacher  while  using  the  GUI 
interface;  Figure  |3]-b  shows  the  first  screen  asking  for  the  shape  selection.  We 
have  designed  simple  and  complex  shapes,  i.e.  circle,  triangle,  hexagon,  rectangle, 
spiral,  porcupin,  bat,  etc.  Figure  |3]-c  shows  the  screen  that  has  been  selected  by 
the  teacher /care  assistant  in  order  to  have  the  visual  feedback.  If  the  FI  key  is 
pressed,  a  circle  is  activated  in  order  to  ask  the  unskilled  user  to  perform  the 
sketch  operation,  if  the  F2  key  is  pressed  a  triangle  is  activated  and  so  on.  This 
graphical  interface  allows  the  teachers  the  possibility  to  switch  between  different 
shapes  and  set  the  modality  task  that  has  to  be  performed,  as  sketching,  hatching 
or  cutting  task. 

Figures  |3]-d  and  |3]-e  show  the  screen  that  has  been  used  during  the  test 
of  the  multimodal  device.  While  the  unskilled  people  are  the  end-users  of  the 
technology,  it  is  possible  to  consider  a  more  person-centered  approach  to  both 
the  evaluation  of  the  multimodal  guidance  and  to  the  research. 

6  Evaluation  of  the  System  by  the  Care  Assistants 

In  our  study  we  have  used  a  qualitative  approach  to  evaluate  the  ergonomics  of 
the  system,  and  to  reach  the  overall  impression  from  users  and  their  assistant. 


Multimodal  Guidance  System  for  Improving  Manual  Skills  in  Disabled  People  231 


Fig.  3.  The  concept  from  the  teacher  and  care  assistant  side 


We  decided  to  interpret  data  as  complementary  to  the  obtained  quantitative 
data.  Table  [T]  shows  the  qualitative  observation  of  the  users. 

One  of  the  major  problems  of  the  qualitative  analysis  is  the  challenging  valida¬ 
tion  of  the  theories  that  rise  from  the  recorded  impressions.  Nevertheless,  those 
kind  of  information  could  be  an  important  index  regarding  the  well-functioning 
of  the  system  and  the  possible  further  improvement.  The  overall  impression  is 
positive.  The  qualitative  data  have  been  obtained  by  two  ways.  For  points  1,4 
and  5  reported  in  Tabled] we  directly  asked  to  users  or  to  the  educators  opinions 
and  suggestions.  For  the  points  2  and  3  we  obtained  data  observing  the  users  be¬ 
havior.  The  system  helping  effect  has  been  found  very  interesting  and  rewarding 
by  the  users,  that  have  been  able  to  focus  their  attention  on  the  task. 


7  Sound  Interaction 

The  sound  feedback  of  the  multimodal  haptic  device  gives  the  possibility  to  play 
metaphoric  sounds  while  the  user’s  interact  with  the  system  .  These  metaphoric 
sounds  provides  information  to  the  disable  people  according  to  the  type  of  task 
performed.  In  fact,  once  the  sound  feedback  is  enabled,  the  sound  feedback  gives 
the  following  information: 

—  Metaphoric  sound  A,  if  the  stylus  pen  is  not  located  directly  on  the  shape. 
This  sound  is  a  kind  of  warning  alarm  and  means  that  the  user’s  pen  is 
located  quite  far  from  the  haptic  shape. 

—  Metaphoric  sound  B,  is  played  when  the  velocity  of  the  stylus  pen  is  higher 
than  an  specific  value.  Also  in  this  case,  the  sound  is  rendered  as  a  warning 
alarm,  and  is  activated  when  the  user’s  pen  goes  too  fast  in  the  sketching, 
hatching  and  cutting  tasks. 
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Table  1.  Evaluation  of  the  Multimodal  Guidance  System 


Main  questions  Results 

Comfort  Is  the  system  comfort-  All  the  participants  involved  into  the  evaluation 
able  for  the  users?  They  have  been  able  to  use  the  system  in  a  comfortable 
can  use  it  in  a  comfort-  way;  the  portability  and  the  reduced  dimensions  al- 
able  position?  low  all  the  subjects  to  set  their  position  in  the  most 

comfortable  way  while  they  performed  the  task. 

Interest  Is  the  system  able  to  in-  For  all  the  participants  the  system  has  been  a  great 
terest  and  to  focus  the  source  of  interest.  In  the  evaluation  phase,  all  of 
user  attention  on  the  them  have  been  able  to  complete  the  task  without 

task?  distraction. 

Usability  Is  the  system  intuitive?  The  majority  of  the  subjects  started  to  interact 

whit  the  system  in  a  correct  way,  grasping  the  sty¬ 
lus  and  moving  it  without  any  instruction. 

Utility  Is  the  system  useful  in  For  all  the  educators  the  system  constitutes  a  valid 
the  educators  opinion?  help  to  their  work,  interesting  and  helping  subjects 
in  the  drawing  and  coping  task. 

System  Provide  the  system  all  In  the  educators  opinion  more  important  forces  are 

com-  the  cues  needed  for  a  needed,  depending  on  the  user  deficit.  Moreover,  an 

pleteness  good  performance  of  the  interactive  and  funny  interface  has  been  suggested 
task?  in  order  to  involve  more  the  subjects  in  the  task. 


8  User  Test 

The  user  test  has  been  performed  involving  6  subjects  (4  males)  aged  18  to  40 
years.  Down  syndrome  and  mental  retardation  affected  all  participants.  In  this 
experiment  a  brief  familiarization  has  been  offered  to  the  participants  and  it  has 
been  asked  to  perform  a  task  that  involved  a  combination  of  visual,  haptic  and 
sound  feedback  in  order  to  design  a  circle  with  100  mm  of  radius.  Figure  |4}a 
shows  the  tracked  motion  of  an  spiral  shape  without  haptic  and  sound  feedback, 
while  Figure  0}b  shows  the  tracked  motion  for  the  same  disable  people  with 
the  haptic  and  sound  feedback.  Figure  shows  the  tracked  motion  of  a  circle 
without  haptic  and  sound  feedback  while  Figure  0]M  shows  the  tracked  circle 
with  the  haptic  and  sound  feedback. 

In  order  to  systematically  assess  the  contribution  of  the  haptic  and  sound 
feedback  we  computed  the  error  between  the  radius  of  two  circles  as  reported 
from  Figures  0]-e  and  0]T.  Results  showed  that  the  error  significantly  decreases 
(Wicloxon  rank  sum  test,  p  0.05)  when  subjects  were  guided.  For  what  con¬ 
cerns  the  evaluation  activities  related  to  the  developed  prototype  of  the  multi¬ 
modal  guidance  system,  we  have  decided  to  evaluate: 

1.  The  applicability  of  the  multimodal  guidance  system  to  the  learning  envi¬ 
ronment. 

2.  Gaining  feedback  on  the  development,  improvement  and  overall  technical 
assessment  of  the  multimodal  guidance  device,  and  suggesting  recommenda¬ 
tions  for  functional  changes  for  subsequent  prototypes. 
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a)  Haptic  +  Sound  feedback  "Off"  b)  Haptic  +  Sound  feedback  "On"  -  e)  Results  haptic  +  Sound  feedback  "Off" 


-0.15  -0.1  -0.05  0  0.05  0.1  0.15  -0.15  -0.1  -0.05  0  0.05  0.1  0.15 

C)  Haptic  +  Sound  feedback  "Off"  d)  Haptic  +  Sound  feedback  "On" 


UPl  UP2  UP3  UP4  UP5  UP6 


f )  Results  haptic  +  Sound  feedback  "On" 


UPl  UP2  UP3  UP4  UP5  UP6 


-0.2  -0.1 


0.1  0.2  0.3  -0.15  -0.1  -0.05  0  0.05  0.1  0.15 


Fig.  4.  User’s  test  results 


9  Conclusion 

The  results  of  our  study  showed  that  the  multimodal  guidance  system  help 
people  during  manual  tasks  by  means  of  using  haptic  and  sound  technology. 

The  opportunity  to  create  a  haptic  system  that  would  make  real  difference  in 
disable  peoples  life  appeared  to  be  a  highly  motivating  factor.  We  are  currently 
performing  an  evaluation  with  unskilled  people  in  order  to  measure  their  learning 
improvements  in  2D  operations  skills.  Results  show  that  the  effect  of  using  the 
haptic  cutting  system  increases  the  accuracy  in  the  tasks  operations.  We  can 
resume  that  the  system  leads  to  the  satisfaction  of  the  following  objectives: 

—  The  force  feedback  enhances  the  interaction  between  the  user  and  the  phys¬ 
ical  template. 

—  The  sound  feedback  as  and  additional  channel  information  in  order  to  per¬ 
forming  the  2D  tasks  according  with  the  correct  velocity  required  to  sketch, 
hatch  and  cut  the  foam. 
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